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SCOPE  AND  OBJECTIVES  ! 

y  ^ 

The  basic  aim  of  the  Group  Judgment  Technology  project  is 
the  development  of  improved  procedures  for  the  use  of  expert 
judgment  as  an  aid  to  government  and  industrial  decisionmaking. 
The  area  of  potential  techniques  is  wide,  extending  from  simple 
polling  of  groups  of  experts  to  the  complex  and  structured  inter¬ 
actions  involved  in  simulation  and  gaming  exercises. 

Similarly,  the  factors  that  determine  the  effectiveness  of 
group  judgment  procedures  for  decisionmaking  cover  an  equally 
broad  range.  No  comprehensive  taxonomy  for  these  factors  has 
been  developed.  Among  the  most  important  factors  are:  (a)  the 

type  of  subject  matter  appropriate  for  group  judgment  procedures, 
;i  (b)  the  characteristics  that  determine  expertise  and  their 

measurement,  (c)  the  aspects  of  group  interaction  that  affect 
performance,  (d)  the  structure  of  the  estimation  task  posed  to 
the  group,  (e)  the  role  of  external  sources  of  information,  (f) 
the  physical  environment  of  the  exercise,  including  communication 
facilities . 

PREVIOUS  EXPERIMENTAL  RESULTS 

Previous  experiments,  including  the  ARPA-sponsored  series 
at  Rand  and  a  number  of  experiments  by  social  psychologists  over 
the  past  50  years,  have  shed  considerable  light  on  some  of  these 


factors.  The  appended  list  of  references  contains  tho  most 
relevant  studies.  Major  findings,  many  of  which  are  based  cn 
ARPA-supported  Delphi  studies,  include:  Delphi  procedures  are 
most  appropriate  where  the  best  information  available  is  the  judg¬ 
ment  of  experts,  i.e.,  where  good  data  and  well-validated  theories 
do  not  exist,  and  where,  for  problem  solving  tasks,  there  is  no 
objective  way  to  monitor  progress  toward  a  solution.  This  excludes 
a  large  number  of  cognitive  tasks  such  as  solution  to  mathematical 
problems,  state-of-the-art  design  of  equipment,  and  the  like. 

A  large  number  of  experiments  have  shown  that  for  the 
appropriate  judgmental  tasks,  there  are  large  advantages  to 
pooling  the  estimates  of  a  group  (*n  heads  are  better  than  one1* 
rule).  These  advantages  include:  the  group  response  will 
generally  be  more  accurate  than  a  representative  individual 
response,  and  the  dispersion  of  the  group  responses  is  a  valid 
indicator  of  the  accuracy  of  the  group  (1,2,3).  Most  of  the 
desirable  formal  properties  of  group  judgment  carry  over  to 
value  judgments  (goals,  objectives,  priorities).  (4) 

Studies  evaluating  the  usefulness  of  group  judgments  on 
highly  uncertain  questions  for  "real-life*  decisionmaking  are 
rare.  Dr.  John  Williamson  at  Johns  Hopkins  has  been  investi¬ 
gating  the  usefulness  of  Delphi  procedures  for  generating 
information  concerning  the  state  of  health  of  the  genoral  public 
as  inputs  to  HEW  policy  formulation  (5).  Most  of  the  informa¬ 
tion  available  on  this  question  consists  of  intuitive  judgments 
by  decisionmakers  on  whether  Delphi  studies  conducted  in  their 
organisations  were  "useful". 
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Information  on  the  characteristics  of  expertise  is  some¬ 
what  fragmentary.  The  two  most  significant  components  appear 
to  be  amount  of  relevant  background  information  and  estimation 
skill.  Low ,  but  statistically  significant  correlations  have 
bssn  found  between  accuracy  of  estimates  on  short-range  fore¬ 
casting  questions  and  scores  on  standard  achievement  tests  [6] 
as  well  as  with  scores  on  standard  intelligence  tests  [7].  A 
conceptual  level  score  (Paragraph  Completion  Test)  appears  to 
be  related  to  dagrae  of  cartainty  (spread  between  low  and  high 
estimates)  of  respondents  (8).  There  is  some  evidence  that 
estimation  skill  can  be  improved  with  short  training  sessions 
1 9 1 .  Self-rated  coaipetence  (on  a  given  question)  has  shown  small 
but  statistically  significant  correlation  with  estimation  accuracy 
(10).  There  is  some  evidence  that  performance  is  related  to 
'cognitive  style*  e.g.,  students  from  the  hard  sciences  perform 
less  well  than  thorns  from  the  social  sciences.  In  general, 
women  perform  less  well  on  estimation  tasks  than  men  (11).  More 
specifically  psychological  parameters  such  as  motivation,  personality 
traits,  and  task  perception  have  received  only  cursory  attention. 

A  cognate  practical  issue  relates  to  the  selection  process 
for  experts  for  a  specific  study.  The  usual  selection  process 
(nomination  by  poors)  is  probably  subject  to  a  number  of  biases, 
including  the  tendency  of  individuals  to  nominate  experts  whose 
opinions  agree  with  their  own  (12).  Bias-reduction  techniques 
have  been  studied  as  part  of  a  projected  extension  of  the 
Johns  Hopkins  study,  but  have  not  yet  been  tested  experimentally. 
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Most  of  the  relevant  work  on  Delphi  procedures  has  been 
aimed  at  identifying  improved  methods  of  group  interaction. 

The  early  emphasis  on  anonymity  of  response,  iteration,  and 
controlled  feedback,  was  based  on  intuitive  "fixes"  to 
informally  observed  problems  with  face-to-face  interaction — 
biasing  effects  of  dominant  members,  group  pressure  and  band¬ 
wagon  effects,  irrelevant  communication  ("noise").  Experiments 
comparing  anonymous  feedback  interaction  with  face-to-face 
interaction  conducted  at  Rand  in  1167-68  indicated  a  slight 
superiority  for  the  anonymous  feedback  mode  [11]  .  Subsequent 
experiments  have  shown  that  much  larger  improvements  can  be 
obtained  with  other  parameters — group  size,  additional  informa¬ 
tion,  selection  of  more  competent  subgroups  on  the  basis  of  self- 
ratings,  statistical  compensation  for  biased  estimates  (e.g. 
use  of  the  63rd  percentile  instead  of  the  median) .  Two  recent 
results  are  that  s  statistical  group  response  is  an  effective 
way  to  pool  diverse  information  within  the  group  (without 
information  transfer)  and  that  a  basic  feature  of  the  group 
estimate,  the  ratio  of  log-error  to  the  standard  deviation,  is 
invariant  over  wide  ranges  of  relevant  input  information  [13). 
These  may  open  up  an  entirely  new  perspective  on  the  significance 
of  group  interaction,  e.g.,  that  part  of  the  poor  showing  of  face- 
to-face  interaction  is  due  to  the  minor  effect  on  accuracy  of 
information  transfer. 

Basic  understanding  of  the  estimation  process  itself  is 
spotty  (some  of  this  will  be  treated  in  Section  3)  .  A  number 
of  significant  features  of  the  process — log-normal  distributions 
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of  responses,  general  tendency  to  underestimate  uncertain 
numbers,  short  optimum  response  time,  nonlinear  increase  in 
error  with  size  of  number  estimated — have  been  relatively 
well  established  by  the  experimental  data,  but  have  not  yet 
received  satisfactory  explanations. 

Other  components  of  the  structure  of  the  estimation 
task  that  have  received  attention  are  the  feasibility  and 
usefulness  of  probability  estimates  [14),  the  relation  between 
question  length  and  a  measure  of  precision  of  response  [15] . 

The  role  of  external  sources  of  information  as  feed-in 
to  a  Delphi  exercise  has  been  studied  in  a  series  of  experiments 
at  Rand.  These  have  indicated  high  improvement  in  accuracy  as 
a  consequence  of  feed-in  of  additional  "hard"  factual  information, 
and  a  decreasing  return  for  each  additional  piece  of  information 
[13]  . 

Experimental  study  of  the  role  of  the  environment,  including 
the  characteristics  of  the  communication  system  has  not  been 
pursued  for  the  kind  of  subject  matter  appropriate  for  Delphi 
treatment.  There  is  a  fairly  large  literature  dealing  with 
these  topics  for  related  tasks  (e.g.,  Bavelas,  Guetzkow  and  Simon; 
and  Shure,  Rogers,  et  al . #  experiments  with  communication  nets  for 
well-defined  problem  solving  tasks)  [16] . 
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SCOPE  OF  APPLICATIONS 

Although  the  amount  of  effort  going  into  experimental 
evaluation  of  group  judgment  procedures  is  small,  the  amount 
of  effort  being  expended  in  applications  is  quite  large,  and 
is  still  expanding  rapidly.  The  major  applications  have  been 
in  the  industrial  area  with  the  forecasting  of  long-range 
technological  developments  as  an  aid  to  industrial  planning. 

The  list  of  industrial  contractors  that  have  initiated  studies 
of  this  sort  is  extensive.  It  includes  companies  such  as  TRW, 
Smith,  Kline  and  French,  IBM,  Xerox,  General  Motors,  McDonnel- 
Douglas,  Martin-Marietta,  and  many  others.  In  a  recent  survey 
of  long-range  forecasting  activities  in  the  United  States  by 
John  McHale  at  the  Center  for  Integrative  Studies  at  the 
State  University  of  New  York  at  Binghamton,  [17]  Delphi  was  listed 
as  the  second  most  utilized  technique  for  long-range  studies 
by  industry,  where  well  over  40%  of  the  organization  reported 
that  they  had  made  use  of  it.  The  same  ratio  was  observed  in 
a  very  large  survey  of  industrial  organizations  and  research 
organizations  conducted  by  Marvin  Cetron.  Out  of  about  a 
thousand  returns,  something  of  the  order  of  40%  said  that  they 
had  utilized  Delphi  for  one  type  of  study  or  another.  The 
Institute  for  the  Future,  founded  by  Olaf  Helmer,  has  in  the 
past  three  years  conducted  roughly  fifteen  extensive  Delphi 
studies  for  both  industry  and  state  governments,  dealing  with 
long-range  forecasts  of  technological  and  social  events. 

The  activity  has  spread  to  practically  every  country 
on  the  globe.  There  have  recently  been  a  number  of  reports 
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of  studies  from  behind  the  iron  curtain.  One  of  the  most 
impressive  of  the  foreign  efforts  is  the  study  recently 
completed  for  the  Japanese  government  which  utilized  4000  of 
their  top  scientists  and  engineers  to  generate  technological 
and  social  developments  as  inputs  for  government  planning. 

They  are  now  in  the  middle  of  Phase  II  of  that  study,  in  which 
they  are  attempting  to  answer  the  question  how  the  set  of 
forecasts  can  be  brought  to  bear  on  the  design  of  government 
policies. 

In  the  United  States  the  federal  government  has  been  a 
little  slow  to  initiate  Delphi-type  studies.  But  at  the  present 
time  there  are  perhaps  30  studies  by  different  government  agencies 
which  are  in  process  or  have  recently  been  completed.  The  CIA 
has  at  least  four  activities  under  way;  one  is  a  study  of  manpower 
requirements  and  training;  another  is  a  study  of  Soviet  objectives 
for  the  Future  Threats  branch;  Delphi  is  being  esq»loyed  as  a 
component  of  their  advanced  analyst  training  course,  and  is  being 
investigated  for  applications  to  estisuites.  DIA  last  year 
conducted  a  large-scale  allocation  exercise  using  Delphi  to — among 
other  things — rate  a  set  of  pertinent  objectives.  The  Naval 
Ordinance  Laboratory  recently  completed  a  rather  extensive  study 
of  future  undersea  requirements.  All  three  of  the  services 
have  several  studies  under  way  involving  items  such  as  evaluation 
of  tactical  targets  and  reliability  of  future  weapons  systems. 
Nonmilitary  government  agencies  which  have  employed  Delphi 
studies  include  the  Office  of  BMrgency  Planning,  the  Coast  Guard, 
Department  of  Transportation,  Internal  Revenue  Service,  and  the 


Forestry  Service. 

The  area  of  subject  matters  to  which  Delphi  studies  have 
been  applied  has  also  expanded.  Topics  that  have  been  studied 
include  items  such  as  the  reliability  of  hardware  and  equipment 
still  in  development,  the  rating  of  corporate  objectives, 
acceptable  weather  conditions  for  paradrops  operations,  and 
identification  of  potential  educational  innovations.  An 
interesting  merging  of  Delphi  technique  with  public  opinion 
survey  techniques  has  been  used  by  a  number  of  research  centers 
for  the  generation  of  objectives  for  elementary  education  [18]. 

In  fact  it  is  difficult  to  find  any  area  of  direct  interest  to 
decisionmakers  where  Delphi  has  not  been  tried. 

PROPOSED  RESEARCH  AREAS 

All  of  the  factors  listed  under  SCOPE  AND  OBJECTIVES  warrant 
further  study.  Eowever,  since  the  primary  aim  of  the  present 
project  is  primarily  technological-*!.#,  the  development  of 
Improved  procedures— some  of  the  potential  areas  of  research  look 
more  promising  than  others  with  respect  to  likelihood  of  fairly 
immediate  return  in  experimental  results  and  usefulness  of  those 
results  for  practical  applications.  In  addition,  the  special 
capabilities  of  CCR8  opens  opportunities  for  the  development  of 
saich  more  effective  group  judgsient  processes  than  have  been  employed 
up  to  now.  The  areaa  of  research  that  appear  moat  promising  are: 

1.  Experimental  Studies  of  Delphi  Methodology 

Several  aspects  of  the  estimation  process  and  the  formulation 
of  group  responses  are  of  direct  importance  for  applied  studies 


of  the  sort  now  being  conducted  in  government  and  industry. 

These  include: 

(a)  Combination  of  Accuracy  Measures.  In  previous  experi¬ 
ments  ,  the  dispersion  of  the  group  distribution  of  responses, 
and  the  average  of  individual  self-ratings  of  competence  have 
been  demonstrated  to  be  valid  indicators  of  the  accuracy  of 
group  response  (median) .  These  studies  have  dealt  with  responses 
to  individual  question.  In  practical  applications,  group  judg¬ 
ments  are  obtained  on  a  complex,  interacting  set  of  questions, 
where  the  responses  are  combined  to  produce  conclusions  of 
interest  to  decisionmakers.  What  little  evidence  is  available 
suggests  that  the  dispersion  and  self-ratingm  do  not  combine  in 

a  normal  statistical  fashion.  The  experiments  will  involve 
obtaining  answers  to  a  given  set  of  questions  in  several  ways — 
direct  estimation,  estimation  of  relevant  factors  and  computing 
the  answer  from  these,  combined  estimation  of  questions  and 
relevant  factors.  The  data  will  give  a  clearer  indication  of 
the  way  in  which  the  accuracy  indices  should  be  combined  in 
applications. 

(b)  Group  Probability  Judgments.  Experiments  to  date  indi¬ 
cate  that  it  is  feasible  for  subjects  to  produce  probability 
distributions  as  responses  to  uncertain  questions,  and  that 

a  summed  distribution  will  score  higher  on  a  probabilistic  scoring 
system  than  on  a  distribution  of  point  estimates  [14].  This 
set  of  experiments  will  extend  the  investigation  to  a  wider  set 
of  questions,  particularly  caBes  of  greater  uncertainty,  and  in 
addition  examine  the  effectiveness  of  self-ratings  for  the 
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selection  of  more  knowledgeable  subgroups  in  the  case  of 
probability  judgments. 

(c)  Level  of  Detail.  In  most  applied  exercises  there  is 
a  tendency  to  "unpack”  decision  problems  to  a  high  level  of 
detail.  Group  judgments  are  then  obtained  on  the  many  elements 
of  the  detailed  analysis.  There  is  no  good  experimental 
evidence  that  this  is  either  the  most  effective  way  of  obtaining 
the  most  accurate  answers  to  questions  which  are  directly  relevant 
to  decisionmakers'  interests  nor  that  this  is  the  most  efficient 
way  to  use  the  knowledge  of  expert  advisors.  The  experiments  will 
test  the  accuracy  of  answers  generated  by  models  at  several  levels 
of  detail  to  see  if  there  is  some  optimum  level,  and  to  identify, 
if  possible,  the  characteristics  of  the  problem  on  which  the 
optimum  level  depends. 

(d)  Bias  Correction  Techniques.  There  appears  to  be  a 
general  tendency  for  respondents  to  underestimate  uncertain 
numbers.  Over  a  very  large  number  of  responses  obtained  in 
previous  experiments,  an  average  improvement  of  about  30  percent 
in  the  log  error  is  obtained  by  using  the  63rd  percentile  as 
the  group  response  rather  than  the  median.  The  error  in  this 
case  appears  as  a  biasi  it  is  much  larger  than  would  be  expected 
from  sampling  theory  alone.  Me  are  comparing  several  models 

of  the  estimation  process  with  the  data  to  see  if  a  firm 
theoretical  base  can  be  identified  for  the  63rd  percentile 
correction  factor. 
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2 .  Analytic  Developments 

In  order  to  make  Delphi  techniques  more  responsive  to  the 
needs  of  practitioners,  several  analytic  capabilities  require 
extension.  Three  which  are  of  immediate  practical  value  are: 

(a)  Extension  of  Probabilistic  Scoring  Systems.  Probabilistic 
scoring  systems  are  among  the  more  powerful  tools  for  investigating 
both  individual  and  group  probability  judgments.  To  date  they  have 
been  devised  for  a  relatively  simple  type  of  judgment — namely  the 
single  forecast  with  either  a  probability  distribution  over  a 
given  quantity,  or  a  finite  set  of  complete  and  exhaustive  alterna¬ 
tives.  To  be  applicable  to  more  complex  types  of  judgment,  the 
formalism  must  be  extended  to  other  forms  of  probability  statements, 
especially  to  the  case  of  relative  probabilities. 

(b)  Cross-impact  Analysis.  One  of  the  more  promising 
procedures  for  long-range  forecasting  of  technological  and  social 
events  is  cross-impact  analysis — a  technique  for  taking  into 
account  the  interrelations  of  events  in  assessing  their  probability 
of  occurrence.  As  developed  at  the  Institute  for  the  Future,  the 
technique  has  suffered  from  logical  flaws  that  have  made  application 
to  the  technique  to  practical  forecasting  studies  questionable. 
Considerable  progress  in  furnishing  a  sound  logical  foundation  for 
the  technique  has  been  made  in  recent  work  at  Rand  [19]  .  This 
work  will  be  pursued,  both  to  furnish  a  meaningful  decision 
structure  for  applications,  and  to  furnish  a  useful  framework  for 
later  experiments  with  on-line  model  construction. 

(c)  Dimension-reduction  Techniques.  One  of  the  basic 
difficulties  which  arises  in  application  is  the  "point-of-view" 
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problem.  For  most  decision  situations,  each  expert  has  "in  the 
back  of  his  mind"  a  crude  model  consisting  of  a  blurred  list  of 
the  most  important  factors  as  he  sees  them,  along  with  a  rough 
feeling  for  the  interactions  among  the  factors.  These  "models" 
can  differ  widely  among  experts — expressing  their  specialized 
backgrounds  and  personal  cognitive  styles.  A  Delphi  exercise  can 
be  used  to  elicit  these  partial  models,  but  in  order  to  aggregate 
them  into  a  coherent  "group  model"  rather  powerful  techniques  of 
collation  and  clustering  are  required.  Multidimensional  scaling 
and  hierarchical  clustering  routines  have  proved  to  be  useful 
for  elementary  aggregation  of  panel-generated  lists  of  corporate 
objectives  for  example;  but  these  techniques  need  augmentation  in 
several  directions  before  they  can  be  considered  satisfactory  for 
on-line  modeling.  In  particular,  there  is  a  need  for  integrated 
display  and  group-editing  procedures.  The  basic  capabilities 
probably  already  exist  in  the  CCBS  system,  but  a  fair  amount  of 
experimentation  will  be  required  to  configure  procedures  for 
efficient  use  by  expert  groups. 

3.  Extending  the  Scope  of  Delphi  Capabilities 

Up  to  now,  experimentation  has  mainly  been  concerned  with 
elementary  group  judgments,  i.e.,  with  the  formulation  of  group 
responses  on  single,  well-defined  questions.  In  practice, 
decision  problems  involve  complex,  interrelated  judgments.  To 
make  the  benefits  of  tho  experimental  program  of  maximal  value  to 
decisionmakers,  it  is  necessary  to  extend  the  evaluation  of 
procedures  of  group  judgment  that  are  appropriate  for  more 
realistic  (and  hence  more  complex)  decision  problems. 
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(a)  Integrating  Delphi  and  Decisionmaking  Structures.  It 
is  very  likely  that  the  volume  of  Delphi  studies  for  a  wide 
variety  of  decision  problems  will  increase  rapidly  over  the  next 
few  years  in  state  and  local  governments ,  in  the  federal  government 
and  in  research  institutions.  There  will  probably  be  a  shake-out 
with  regard  to  the  elementary  type  of  Delphi  study  exemplified  by 
the  identification  of  technological  developments  in  a  given  area 
and  estimates  of  the  time  of  occurrence  and  some  estimate  of  the 
relative  importance.  By  and  large  this  kind  of  study  has  turned 
out  to  be  of  peripheral  interest  to  decisionmakers  and  not  directly 
relevant  to  policy  formulation.  The  most  successful  studies  have 
been  those  which  have  been  formulated  directly  within  a  decision¬ 
making  context  where  the  implications  for  policy  were  defined 
ahead  of  time.  Representative  cases  are  the  use  of  Delphi  by 
Corning  Glaes  (20]  to  select  development  projects  and  the  DIA 
study  mentioned  earlier.  There  will  be  a  general  increase  in 
studies  of  the  sort  where  the  Delphi  exercise  is  an  integral  part 
of  the  decision  process,  involving  the  formulation  of  corporate 
or  agency  goals,  the  identification  of  major  policy  options 
and  the  estimate  of  the  impact  of  each  policy  option  on  the  set 
of  goals.  Such  a  structure,  for  example,  would  find  a  direct 
application  in  budget  allocation  problems. 

Experiments  to  evaluate  the  appropriateness  of  Delphi 
procedures  for  generating  and  rating  objectives  have  been 
conducted  with  favorable  outcomes  (4).  Relevant  experiments 
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with  a  more  complete  decision  structure  in  which  potential 
careers  wore  rated  against  a  set  of  personal  objectives  also 
indicate  that  group  ratings  on  objectives  are  more  reliablo 
predictors  of  individual  choice  than  individual  ratings  (21]. 

However,  careful  experiments  to  pin  down  the  range  of  appropriate¬ 
ness  have  not  been  conducted  to  date. 

(b)  On-line  Model  Construction.  One  of  tho  unoxploitod 
capabilities  of  Delphi  is  its  use  for  the  generation  of  modols. 

As  noted  under  the  section  on  dimension-reduction  techniques, 
experts  usually  have  partial  models  of  the  decision  situation 
"in  the  backs  of  their  minds".  These  can  be  elicited  by  Delphi 
techniques,  and  an  aggregated  group  model  derived.  Por  applied 
studies  it  is  not  necessary  that  a  completely  formalized  mathematical 
structure  be  generated.  The  group  acts  as  a  kind  of  model  in  that 
it  can  integrate  an  extensive  amount  of  information  on  its  own. 

What  is  required  is  that  the  structure  of  the  decision  problem  be 
specified  sufficiently  that  subsequent  group  estimates  are 
directly  relevant  to  the  pertinent  decision  issues.  This  was  in 
outline  the  process  pursued  in  the  first  Delphi  study  at  Rand  that 
used  systematic  group  procedures  with  Iteration  and  feedback  (22]. 

At  present,  if  this  process  is  conducted  by  the  usual  questionnaire 
technique- -especially  with  geographically  dispersed  panels — time 
lags  are  likely  to  be  excessive.  On-line  interaction  can 
drastically  reduce  time  lags,  and  at  the  same  time  allow  preliminary 
computations  with  initial  versions  of  the  model  as  a  basis  for 
subsequent  revisions.  Needed  are  techniques  for  collating  and 
structuring  the  natural  language  responses  typically  obtained 


15 


in  tho  oponing  rounds  of  a  modol  identification  oxercie«». 

(c)  Improved  Interface  with  Formal  Decision  Models.  At 
present,  formal  decision  models  such  as  optimal  budget  allocation 
techniques,  cost-benefit  models  and  the  like  do  not  mesh  well 
with  group  judgment  techniques.  The  primary  roason  is  that 
with  Delphi  procedures  it  is  generally  feasible  to  obtain  estimates 
concerning  a  wide  range  of  parameters  not  normally  included  in 
the  formal  aw>dels,  especially  items  of  the  sort  usually  labelled 
"intangibles".  Techniques  are  needed  to  extend  the  formal  models 
to  include  this  wider  range  of  information. 

4.  Adapting  the  Capabilities  of  CCB8 

The  extensive  on-line  facilities  available  through  CCBS 
will  make  experimentation  with  cos^lex  decision  structures  much 
more  feasible  than  in  the  past.  In  addition,  these  facilities 
can  operate  as  a  test  bed  for  on-line  decisionmaking  procedures 
of  a  sort  likely  to  become  widespread  during  the  next  decade  in 
both  governmental  agencies  and  Industrial  corporate  management. 

The  advantages  of  on-line  computer  systems  for  group  interaction 
are  clearly  quite  large.  They  include,  among  others,  the  very 
great  time  compression  that  can  be  achieved,  along  with  the 
possibility  of  geographic  dispersion  of  the  panel.  In  addition 
there  is  the  ability  to  deal  with  very  large  volumes  of  data,  which, 
for  example,  allows  the  possibility  of  large  panels  of  respondents, 
but  also  large  numbers  of  responses  from  each  panelist.  There 
also  is  the  possibility  of  the  group  interacting  with  extensive 
data  banks  and  the  possibility  of  sophisticated  computation  as 
—  * anaaino  exercise.  The  elementary  Delphi 
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laboratory  at  Hand  was  on**  exploratory  activity  in  thin  .iron. 

Murray  Turoff  at  ot'P  has  tried  an  oxploratnry  exercis**  with  a 
geographical ly  dispersed  group,  utilizing  a  time-sharing  uystum,  1 2 3 1 
which  had  the  rather  significant  spinoff  that  it  enabled  the  rapid 
creation  of  an  on-line  information  system  to  Implement  the  rocont 
wags/price  freeze.  The  Plato  project  at  tho  University  of  Illinois 
has  explored  soveral  aspects  of  nan-model  interaction  (24).  A 
number  of  other  exploratory  exercises  are  going  on  throughout  tho 
country,  primarily  directed  toward  the  development  of  on-line 
conferencing  systems. 

(a)  Software  for  Delphi  Experimentation.  The  laboratory 
configuration  with  24  stations  and  highly  flexible  data  acquisition, 
analysis  and  coossunication  capabilities  is  ideal  for  Delphi 
experimentation.  Programming  the  requisite  query  routines,  feed¬ 
back  data  displays,  and  intra-round  analysis  should  be  highly 
efficient  using  the  LIS  capability  (25).  In  addition  to  these 
"standard”  elements  of  a  Delphi  exercise  it  will  be  possible  to 
develop  routines  for  collecting  nonnumerical  responses,  ana 
processing  these  in  terms  of  emendation  by  the  group. 

(b)  Interaction  with  Data  Pases.  Previous  experiments  have 
demonstrated  a  large  improvement  in  accuracy  when  precise, 
relevant  information  is  available  at  the  time  the  estimates  are 
being  made.  One  of  the  capabilities  inherent  in  the  CCBS  configura¬ 
tion  is  rapid  access  to  extensive  data  files,  and  on-line  explora¬ 
tion  of  these  via  the  TRACE  system  (261*  Another  is  the  capability 
of  real-time  augmentation  of  such  files  by  direct  input  from 
experts,  either  individually,  or  as  group  judgments.  Thus  the 
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files  sllow  foed-in  of  information  from  external  sources,  inter¬ 
change  of  information  among  the  panelists,  and  mutual  evaluation 
of  the  information. 

(c)  On-line  Delphi  Exercises.  Present  questionnaire 
techniques  for  conducting  Delphi  studies  result  in  long  lag 
times  (typically  up  to  six  months  for  a  three-round  study).  Part 
of  the  lag  time  results  from  delays  in  the  mail,  part  from 
processing  delays  (key-punch,  batch  processing,  etc.).  Most  of 
this  delay  can  be  eliminated  by  using  an  on-line  system  for  data 
acquisition  and  processing.  On-line  Delphi  studies  with 
geographically  dispersed  panels  can  be  run  shortly  after  the 
elementary  Delphi  software  is  implemented.  However,  these  would 
be  relatively  simple  exercises  with  preprogrammed  questions.  To 
conduct  more  sophisticated  studies  in  which,  e.g.,  the  panel  could 
formulate  its  own  questions  requires  some  of  the  additional 
capabilities  discussed  above.  On-line  exercises  with  geographically 
dispersed  panels  will  be  possible  with  much  greater  economy  and 
more  extensive  capabilities  when  CCBS  is  integrated  into  the  ARPA 
net. 

5.  Studies  of  the  Estimation  Process 

In  addition  to  the  study  of  complex  judgments,  there  is  good 
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reason  to  believe  that  basic  improvements  in  Delphi  procedures 
can  be  made  by  obtaining  a  clearer  understanding  of  the  estimation 
process  itself.  The  intent  here  is  to  identify  and  "calibrate" 
the  elements  or  "phases"  of  the  process  of  producing  a  numerical 
estimate.  It  seems  probable  that  when  a  subject  is  requested  to 
furnish  an  answer  to  a  question,  at  least  four  separate  activities 
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take  place:  (a)  ingesting  and  understanding  the  question,  (b) 
memory  search  for  relevant  information,  (c)  evaluation  of  this 
material  with  respect  to  relevance  and  validity,  (d)  generation 
of  a  numerical  estimate  based  on  acceptable  information.  These 
activities  can  be  replicated  in  feedback  loops  several  times 
before  a  final  judgment  is  expressed.  For  example,  evaluation  can 
lead  to  further  search  for  useful  information;  an  unexpected 
estimate  can  lead  to  re-examination  of  the  question — "Did  I  under¬ 
stand  it? — and  the  like. 

The  long-range  interest  here  is  whether  these  activities  can 
be  experimentally  defined,  and  usefully  quantified.  One  basic 
quantification  is  afforded  by  time,  i.e.,  the  time  to  carry  out 
the  activity.  Time  dissection  will  require  experimental  techniques 
that  are  only  partially  developed  at  present;  but  there  is  reason 
to  believe  that  with  the  CCBS  system  available  techniques  will 
evolve  as  data  accumulates. 

Primary  emphasis  in  the  studies  will  be  placed  on  those 
characteristics  of  the  estimation  process  which  are  relevant  to 
the  efficiency  and  accuracy  of  the  estimate.  It  seems  rather 
likely  from  informal  observation  that  much  of  the  information 
generated  by  the  memory  search  activity  is  not  only  irrelevant, 
but  also  biasing.  The  time  vs.  accuracy  result  indicates  that 
something  in  the  process  is  quickly  saturable;  this  could  be  the 
number  of  pieces  of  information  that  can  be  usefully  handled  by  the 
generation  process,  or  simply  the  number  of  items  of  information 
that  can  be  kept  in  mind  at  one  time  (George  Miller's  "Magical 
number  seven") .  Without  prejudging  the  experiments,  it  appears 
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likely  that  understanding  this  saturation  phenomenon  could  load 
to  basic  improvements  in  estimation  procedures. 

Initial  experiments  will  involve  a  more  careful  set  of 
timing  studies ,  where/  to  the  extent  possible,  parts  of  the 
process  will  be  bypassed.  Exploratory  sessions  have  been  run, 
where  the  subjects  are  told  beforehand  the  kind  of  question 
that  is  to  be  asked,  and  respond  to  a  single  word  or  short  phrase. 
Initial  investigations  have  shown  very  short  estimation  times 
when  the  subject  is  instructed  to  write  down  the  first  number  that 
comes  to  mind  as  an  answer  to  the  question.  In  general,  about 
two  seconds  is  required  to  read  the  name  and  form  an  initial 
estimate.  Comparison  runs,  where  the  subject  is  presented  a 
number  and  simply  asked  to  write  down  that  number,  indicate  that 
possibly  one  half  of  the  response  time  consists  in  reading  the 
its*  and  preparing  to  write  down  a  response. 

RESEARCH  PROGRAM 

To  assure  the  achievement  of  these  capabilities,  a  relatively 
long  time  horizon  must  be  assumed.  Three  years  appears  to  be 
about  the  minimum  needed  to  progress  from  the  elementary  procedures 
now  available  to  the  integrated  on-line  decision  process  appropriate 
for  the  later  '70' a.  In  the  research  program  described  below, 
activities  for  fiscal  '73  are  specified  in  some  detail.  Activities 
for  the  following  two  years  are  sketched  with  somewhat  less  detail 
since  they  will  depend  to  some  extent  on  the  results  of  the  first 
years  investigations. 
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1 .  Research  Proposed  for  Fl«cal  *73 

The  research  which  is  proposed  for  the  coming  year  is 
relatively  narrow  in  scope  and  aimed  at  obtaining  experimental 
results  that  will  be  of  immediate  and  direct  use  for  present 
generation  applied  studies.  Thus,  the  activities  will  consist 
mainly  of  setting  up  the  CCBS  facility  for  running  Delphi 
exercises,  and  carrying  out  the  set  of  experiments  listod 
under  Experimental  Studies  of  Delphi  Methodology.  A  small  part 
of  the  effort  (about  10%)  will  be  expended  on  continuing  the 
analytic  work  on  probabilistic  scoring  systems  and  cross-impact 
analysis.  Another  fraction  of  the  effort  will  be  expended  in 
developing  the  on-line  collating  and  group  editing  procedures 
discussed  under  On-Line  Model  Generation  and  relating  these 
to  appropriate  dimension-reduction  computations  (cluster  analysis 
to  begin  with) . 

2 .  Research  Proposed  for  *74,  *75 

The  research  program  for  fiscal  '74  and  '75  will  be  focussed 
specifically  on  developing  integrated  procedures  for  on-line 
decisionmaking  of  the  sort  appropriate  for  industrial  or  governmental 
agency  decisions.  General  features  of  this  process  are:  rapid 
response,  geographically  dispersed  panels,  interaction  with  data 
banks,  and  on-line  modeling.  Wherever  feasible,  development  will 
be  conducted  in  conjunction  with  relevant  substantive  studios,  so 
that  direct  usefulness  of  the  techniques  will  be  apparent. 

The  program  will  consist  of  the  simultaneous  analysis  of  a 
relevant  decision  problem,  and  implementation  of  the  components 
of  the  on-line  decision  process  discussed  above.  A  pertinent 
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example  might  be  the  generation  of  a  decision  modol  for  budgetary 
allocation  across  scientific  fields.  The  exerciso  would  consist 
of  (a)  panel  generation  of  rosoarch  objectives,  (b)  panel  rating 
of  rolativo  importance  of  the  objectives,  (c)  panel  identification 
of  the  major  factors  determining  research  productivity,  (d) 
compilation  of  an  extensive  data  base  on  presently  funded  projects 
structured  according  to  the  factors  identified  in  NcN,  (e)  panel 
rating  of  a  selected  sample  of  projects  in  terms  of  their  contribu¬ 
tion  to  the  objectives  identified  in  "aM,  and  an  overall  rating 
in  terms  of  scientific  value,  (f)  test  of  a  linear  prediction 
model  of  contribution  to  research  objectives  in  terms  of  the 
factors  compiled  in  NdN,  on-line  revision  of  judgments  in  light 
of  the ‘results  of  the  linear  estimation. 

This  exercise  would  generate  the  evaluation  "half"  of  the 
budgetary  model.  The  second  phase  of  the  exercise  would  consist 
of  the  on-line  generation  of  a  set  of  budgetary  alternatives, 
assessment  of  the  impact  of  each  alternative  on  the  factors  "c", 
and  the  computation  of  effect  of  each  alternative  on  the  research 
objectives.  This  exercise  could  be  accompanied  by  a  control 
portion,  in  which  each  panelist  assessed  directly  his  judgment  of 
the  impact  of  each  alternative  on  each  research  objective. 

In  order  to  conduct  the  Delphi  exercise  sketched  above,  almost 
all  of  the  capability  a  mentioned  would  have  to  be  implemented  to 
a  significant  extent. 
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